Discovering the Biomedical Deep Web

The rapid growth of biomedical information in the Deep Web has produced unprecedented challenges for traditional search engines. This paper describes a new Deep web resource discovery system for biomedical information. We designed two hypertext mining applications: a Focused Crawler that selectively seeks out relevant pages using a classifier that evaluates the relevance of the document with respect to biomedical information, and a Query Interface Extractor that extracts information from the page to detect the presence of a Deep Web database. Our anecdotes suggest that combining focused crawling with query interface extraction is very effective for building high-quality collections of Deep Web resources on biomedical topics. 1 Project and System Overview This research tackles two issues: Firstly, a lot of information on the web does not have an explicit URL and is not indexed by traditional search engines. This information is stored in content rich databases that are collectively called the Deep Web [3]. Our system aims to identify these Deep Web databases. Secondly, domain-specific web portals are growing in importance, as they improve usability. Our goal is to build a system that can discover sources for biomedical information in the deep web, while automatically maintaining and updating the information. We choose the biomedical domain for our system due to its importance and the lack of existing portals that contains extensive information on it (especially those available from the Deep Web). The system has three main components: a classifier which makes judgments on pages crawled to decide on link expansion, a focused crawler [1] with dynamically changing priorities governed by the classifier, and a query interface extractor [2] which uses information from the page to determine the presence of a deep web database. 2 Experiments and Results The core part of our experiments is to validate our reasons for using a focused crawler as opposed to a traditional crawler and to measure the success of determining deep web databases that contain biomedical information. The experiments are divided into three parts: 1) Determine the right parameters for the classifier, 2) Measure the effectiveness of the focused crawler to find biomedical data quickly and efficiently, and 3) Locate query interfaces in relevant pages to identify biomedical deep web databases. Measuring the Effectiveness of the Focused Crawler The classifier was trained on a biomedical corpus [4] using TFIDF to produce an accuracy of 90.4% and an F-measure of 94.1% which indicates that the system identifies Discovering the Biomedical Deep Web 617 most positives and negatives accurately but does have a few false results as well. The most crucial evaluation of our focused crawler is to measure the rate at which relevant biomedical pages are acquired, and how effectively irrelevant pages are filtered off from the crawl. Harvest Ratio of Traditional vs Focused Crawler 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 200 400 600 80